Visual Queries: The Foundation of Visual Thinking

There is no visual model of the world in our heads. Over the past few years the phenomena of change blindness and inattentional blindness as well as studies of the capacity of visual working memory all point to the fact that we do not retain much about the world from one fixation to the next. The impression we have of a detailed visual environment comes from our ability to make rapid eye movements and sample the environment at will. What we see at any given instant in time is determined by what we are trying to accomplish. We see what we need to see. If we need to find a path through a crowd we see the openings. If we are trying to find a friend we see the faces. We can think of this process of seeing as the execution of a continuous stream of visual queries on the environment. Depending on the task at hand the brain constructs a visual query and we execute a visual search to satisfy that query. Making visual queries a central concept opens the door to a theory of how we think visually with interactive displays. The process can be thought of as constructing and executing queries on displays. Problem components are formulated into questions (or hypotheses) that can be answered (or tested) by means of pattern discovery. These are formulated into visual queries having the form of search patterns. Visual eyemovement scanning strategies are used to search the display. Within each fixation, active attention determines which patterns are pulled from visual cortex subsystems that do pattern analysis. Patterns and objects are formed as transitory object files from a proto-pattern space. Elementary visual queries can be executed at a rate of 40 msec per simple pattern. Links to non-visual propositional information are activated by icons or familiar patterns, bringing visual information simultaneously into verbal working memory.